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that systematic procedural changes did improve the teachers* 
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who were trained directly had improved their efficiency by the end of 
the year, while teachers trained primarily by means of manuals had 
reduced their efficiency* This suggested that face to face training 
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Abstract 

A series of five studies examined teacher efficiency in employ- 
ing repeated, curriculum-based measurement* The studies involved 
a group of 10 special education teachers in a rural educational co- 
operative, '.he first study established a baseline rate of teacher 
efficiency; the next three studies examined the effect of alternative 
measurement strategies on that efficiency; a final study followed up the 
efficiency of these teachers one year after training and contrasted 
their efficiency with the efficiency of a group of teachers who had 
been trained differently. Single case experimental designs and de- 
scriptive statistics were employed. Results indicated that measure- 
ment activities were time consuming for teachers at first, but that 
systematic procedural changes did improve the teachers 1 efficiency. 
Additionally, the follow-up study revealed that teachers who were 
trained directly had improved their efficiency by the end of the year, 
while teachers trained primarily by means of manuals had reduced their 
efficiency. This suggested that face-to-face training procedures might 
affect initial teacher ef f iciency^ as well as improvement in efficiency 
over time. 



Teacher Efficiency in Continuous Evaluation 
of IEP-Goals 

Since the passage of P.L. 94-142, and with the increasing demand 
for accountability in the schools, educators have been required to 
support empirically their decisions that affect handicapped students. 
In attempting to accomplish this, both practitioners and researchers 
have become increasingly aware of the need for psychometrically sound 
testing that is directly relevant to classroom instruction (Tyler & 
White, 1980). 

Frequent curriculum-based testing along with time-series analysis 
of data has received increased attention in the past decade as a means 
of formulating and documenting decisions that affect handicapped students 
(Jenkins, Dano, & Mirkin, 1979; Lovitt, 1977; White & Haring, 1980). 
Repeated measurement of academic behaviors has several psychometric 
advantages over pre-post administration of standardized achievement 
tests (Fuchs & Deno, 1981). Frequent direct testing has content validity 
with respect: to a student's curriculum and appears to affect positively 
student achievement (Haring & Krug, ,1975; Haring, Maddux, & Krug, 1972; 
Jenkins, Mayhall, Peschka, Townsend, & Krug, 1974; Mirkin, Deno, Tindal, 
& Kuehnle, 1979). 

Unfortunately, teachers commonly report that repeated measurement y 
procedures kre time-consuming (Deno, 1980), and practitioners often fail 
to collect data according to predesignated time schedules (Mirkin, Fuchs, 
Tindal, & Deno, 1981). Therefore, the purpose of this series of studies 
was first to determine how time consuming direct, repeated measurement 
procedures are, and second to explore strategies that might render these 
procedures more time efficient and satisfactory for teachers and that 
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might increase the likelihood that teachers *ill adhere to measurement 
schedules. 

The research presented here was conducted as part of a maj 
study of continuous evaluation of IEP^ goals in a rural Special Educa- 
tion Cooperative, and it is segmented in this report into five studies. 
The first study was designed to establish a baseline rate of teacher 
efficiency in employing repeated measurement both in a simulated and 
in a natural setting. In establishing these baseline rates, teachers 
were taught the measurement procedures and then required to observe 
their own measurement activities in order to increase the number of 
observations sampled and thereby to formulate more reliable estimates 
of teacher efficiency. A by-product of Study 1, consequently, was an 
assessment of whether teachers can reliably observe their own measure- 
ment activities. 

The next three studies in this report examined the effects of 
various logistical arrangements on teacher efficiency" and teacher 
satisfaction in order to determine whether the feasibility of frequent 
measurement can be enhanced. The final study was a follow-up of 
feacher efficiency in the school district in which Studies 1 through 
4 were conducted, as well as a contrast of this efficiency .with that 
of teachers who had been trz.-\ *ed differently in the use of the same 
measurement procedures. , 

Method 

Subjects 

In all five studies, subjects were 10 special education elementary 



resource teachers (2 -male, 8 female), in a rural educational coopera- 
tive who were required by their special education director to partici- 
pate in the studies. Their teachi^experience ranged from 0 to 10 
years. Additionally, in Study 5, a contrast group of six female, 
teachers in a suburban school district volunteered to serve as subjects. 
Teacher Efficiency - 

Teacher efficiency was one of the dependent measures used in alj 
studies. It was categorized into "teacher time" and "student transition 
to task time." 

Teacher time . For this first category, the dependent data reflected 
the amount of time engaged in the measurement of reading, spelling, and 
written expression. For initial observations, the reading measures 
consisted of (a) one-minute sarples of correct words and errors on reading 
passages randomly selected from a basal reader, and (b) one-minute samples 
of correct words and errors on lists of isolated words randomly selected 
from the core list in Basic Elementary Reading Vocabularies (Harris & 
Jacobson, 1972). The spelling measures were one-minute samples of correct 
letters in sequence for words dictated from lists of isolated words 
randomly selected from the core list (Harris u Jacobson, 1972); the 
written expression measures were three-minute samples of words written 
in response to story starters. For subsequent observations, the measures 
were similar, but developed by teachers after they had been trained to do 
so. These measures were based on their own curriculum materials. 

The dependent data for the "teacher time" category were operation- 
alized into four behavior categories (see Observation Recording Sheet 
in Appendix). The first category was " preparation while student was not 
Present." While the student was not present, the teacher found and 



selected student sheets, teacher sheets, graphs, and necessary equip- 
ment (stopwatch, pencils, acetate sheet, grease pencil, etc.). The 
teacher also put away materials while the student was not present. 
To time this preparation, the observer began a stopwatch as the teacher 
first touched materials in preparation for measurement • The observer 
continued to time through minor interruptions, but stopped timing 
for major interruptions such as phone calls, teacher/principal inter- 
ruptions of preparation time, and so on. Timing was done as the teacher 
put away student materials. The observer stopped the timer as the 
teacher finished preparation. At this point, the observer recorded 
the number of students for whom materials had been prepared and put 
away. 

The second* behavior category for "teacher time 11 was "preparation 
while student was present ." While the student was present, the teacher 
found and selected stimulus materials, response sheets, graphs, and 
necessary equipment (stopwatch, pencils, acetate sheet, grease pencil, 
etc.). The teacher also put away materials while the student was present. 
To time this activity, the observer began a stopwatch as the teacher 
first touched materials in preparation for measurement. The observer 
continued to time through minor interruptions but stopped timing for 
major interruptions such as student fights, student temper tantrums, 
principal visits, etc. The observer also timed as the teacher put away 
materials. The observer stopped the timer as the teacher finished 
preparation. Again, the observer indicated the number of students for 
whom materials had been prepared. 

The third behavior category was " directions ." The teacher provided 
instructions to th,2 students for the measurement task. To time this, 



the observer began a stopwatch as the teacher initiated instructions; 
timing was terminated* as the teacher finished giving instructions to the 
student. 

The fourth category was " scoring and g raphing" After administering 
the test, the teacher- scored and graphed student performance. To time 
this, for each academic behavior measured, the observer began a stopwatch 
as the teacher began scoring; the observer stopped the timer immediately 
after performance was graphed. 

Student transition to task time . The second category of teacher 
efficiency reflected student efficiency in transitioning back to 
normal classroom activity.. This was estimated by teachers on a 
self-report questionnaire. (See Student Transition Recording Sheet 
in the Appendix.) On this questionnaire, teachers estimated how long 
it took the studeAt to begin a new activity after completing measure- 
ment and the number of times the teacher reminded the -student to begin 
his/her new activity. 

Teacher Satisfaction 

.o ■ 

Teacher satisfaction was measured using self-report surveys. Two 

types of surveys, designed to tap two "aspect's of satisfaction, were conducted. 

The first measured teacher satisfaction with the. efficiency modifications 

immediately following the experimental phases. The second obtained in- 

fonnai^n "on actual teacher practices several weeks following ex ? erimen 

tal phases. . (See Teacher Surveys in the Appendix.) 

Study. 1; Efficiency of Procedures and 
Reliability of Self-Observation 

Procedure 

Two workshops were held t Q tr/ih teachers. At the first workshop, 



teachers were taught to organize, administer, and score and graph aca- 
demic measures that had been prepared and organized into a "Kit." The 
"Kit" is Part 1 of the Institute for Research on Learning Disabilities 
. (IRLD) Direct and Repeated Measurement Package, designed to train 
teachers to select and implement a repeated measurement system for the 
purpose of monitoring students 1 academic progress .and evaluating the 
success of different teaching strategies. The "Kit" provides the teacher 
with four weeks of i?eas"fementwmaterials in reading, spelling, and 
written expression co be implemented before the teacher selects his/her 
own measurement format. 

For this study the teachers performed several activities using 
(a) the IRLD "Kit," (b) a stopwatch worn around the neck, and (c) infonna 
tion about the age-grade and instructional levels in each doioain for 
a student. The teacher's activities were: 

1. % Preparation of all the measurement materials for that 

day in reading," spelling, and written expression. She/he: 

n (a) Selected all materials required for the measurement 

tasks. 

(b) In pencil, wrote the student's name on the student and 
the teacher sheets. 

(c) For all students, put- these materials in a f olden onto 
which an acetate sheet and grease pencil had been 
permanently attached. 

(d) Placed the folder on the table where' the measures* were 
administered- - - ^ 



Calling the student to the table and measuring the student's 
behavior. ' 

(a) For reading words in isolation, she/he: 

(1). Found the student sheet and placed it facing the 
student. 



(2) Found the teacher sheet, covered it with the acetate 
sheet, and placed it facing the teacher. 

0) JL n u CeSSary, / ead the ««urement directions that 
had been taped to the back of the clipboard. 

^ (4) Administered the task. 

(5) Put away the student sheet. 

(6) Scored and graphed student data. 

(7) Erased the acetate sheet. 

) For reading words in context, she/he: 

U) Itudfnt he StUd6nt Sh6et PlaC6d itfacin 8 the 

v 

(21 Found the teacher sheet, covered it with the acetate 
sheet, and placed it facing the teacher. 

(3) If necessary, read the measurement directi^s that 
had been taped to the back of the clipboard. 

(4) Administered the task. 

(5) Put away the student sheet. 

(6) Scored and graphed student data. 

(7) .Erased the acetate sheet. 
For spelling, she/he: 

(1) Found the student sheet and placed it facing fhe 
student. 6 

(2) Found the teacher sheet, covered it with th,« 
acetate sheet, and placed it facing the teacher. 

(3^ If necessary, read the measurement directions that 
had been taped to the back of the clipboard. 

(4) Administered the task. 

(5) Put away materials, placing the student sheet into 
a folder labeled "to be scored." 

(6) 'Whenever convenient, scored and graphed student data. 
For written expression, she/he: 
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(1) Found the student sheet and placed it facing the 
student. 

(2) If necessary, read t the measurement directions that 
had been taped to the back of the clipboard. 

(3) Administered the task. 

(4) Put away materials, placing the student sheet into 
a folder labeled "to be scored. 11 

(5) Whenever convenient, scored and graphed student data. 
Over five weeks IRLD staff periodically observed teachers in 

these activities during both the workshops (simulated condition) and 
the classroom (natural condition).. During two workshop observations, 
teachers were paired. Alternating rol^s, one teacher played the student 
while the other teacher administered measurement tasks. In the classroom 

c 

observations, the teachers administered the measurement tasks to the 

same- students for several weeks. 

At the Second workshop, teachers were taught to measure their 

6. 

own behavior using the above procedures and a reliability check was - 
made by trained IRLD observers. Then, for- the next two weeks, teachers 
observed their own behavior in the classroom with a reliability check 
conducted one week into their observations. 
Results 

Teacher efficiency was operationalized as time engaged in measure- 
ment. The range and median teacher time were calculated for (a) Meas- 
urement preparation without student present, (b) measurement preparation 
with student present, and (b) the entire administration including direc- 
tions, measurement task (standardized as one minute each for reading 
passages, reading lists, and spelling, and three minutes for written 
expression), and scoring and graphing. 
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Efficiency , A Wilcoxon Matched Pairs Signed Ranks Test was 
applied to the data and did not reveal a statistically significant 
difference between efficiency in the simulated and the natural condi- 
tions. However, teacher total time on the first trial (21 minutes, 
42 seconds) was significantly different (£ < .05) than teacher total 
time for the average of the other three trials (12 minutes, 48 seconds). 

Insert Table 1 about here 

i 

Inspection of Table 1 reveals that efficiency for preparation time 
both with and without the student present followed the same pattern of re 
suits. Additionally, median teacher efficiencies within each trial on 
the different academic areas of measurement were similar for performances 
on directions, measurement task, and scoring/graphing in the two settings 

Reliability . Inter-rater reliability coefficients (shorter ob- 
server time/ longer observer time) were calculated between the teachers 1 
self-observations and the IRLD staff's observations of the teachers. 
Within the simulated setting, the coefficients ranged from .80 to .90 
with a mean coefficient of .86. Similarly, within the natural setting, 
coefficients ranged from .70 to .94 with a mean coefficient of .85. 

Additionally, reliability for various components of the measure- 
ment procedure (i.e.," preparation time, directions, scoring and graphing) 
were all similar, the reliability within each component in both simu- 
lated and natural settings ranging from .75 to .94. Within each con- 
dition, the lowest reliability coefficient occurred in preparation time 
with student present. Within each of the academic areas, the reliabil- 
ity coefficients in both simulated and natural conditions were similar, 
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ranging from .80 to .93. Employing the criterion of .80 as an adequate 
coefficient, reliability in all instances except preparation time with 
the student present in the natural setting was acceptable. 
Discussion 

On the first trial when the procedure was initially taught, total 
teacher time engaged in measurement activities was 21 minutes, 42 
seconds. On the following trial, however, teacher time dramatically 
dropped to 13 minutes, 47 seconds. This difference in teacher time 
between first and second trials appears to indicate that teachers 
quickly gained efficiency in measurement; within one week, they had 
'significantly improved their efficiency by reducing the time required 
for preparation, administration, and scoring and graphing by nearly 50%. 
Nevertheless, an average of 12 minutes, 48 seconds per student (the 
total time for the average of the last three trials) represents a large 
portion of teacher time to spend in measurement activities. Furthermore, 
the efficiency data reported here do not include teacher time engaged 
in reading and analyzing graphs, important tasks if data are to be em- 
ployed meaningfully. Therefore, this measurement format is even more 
time consuming than indicated by the figures reported in this study. 
Given the fact that these procedures appear to be time consuming for 
teachers, Studies 2 and 3 and the single case experiments, investigating 
logistical arrangements that improve teacher efficiency, appear warranted 
if a direct and repeated measurement system is to oe implemented prac- 
tically by teachers. 

Reliability . Teachers reliably measured their own behavior across 
different academic areas, in both simulated and natural conditions, and 
Vithin various components of the measurement procedure, with the possible 
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exception of preparation time with student present in the natural setting. 
Therefore, to sample teacher behavior across occasions in order to better 
estimate teacher efficiency, it appears feasible to rely on teachers' 
self-monitoring of measurement activities. Based on these results, 
teachers' self-observations were employed in the remaining studies. 

Study 2: Efficiency of Prescribed Order 

Procedure 

For one week (Phase A), given one student, teachers administered the 

measurement tasks in whatever order they preferred and each day completed 

the Observation Recording Sheet (see Appendix). For the following week 

(Phase B) given the same student, teachers administered the measurement 

tasks in a prescribed order. The order was designed to allow teachers to 

use the student's response time for the written expression measure (3 

minutes) to score and graph previously administered tasks. The prescribed 

order was determined by the number of tasks administered on a given day. 

On days when two tasks were administered, the teachers: 

— Administered the spelling measure and reading lists in 
any order and scored/graphed whenever they preferred. 

On days when three tasks were administered,- the teachers^ 



(a) Administered the spelling measure, but did not score and 
graph at that time. 

(b) Administered the reading lists, but did not score or graph 
at that time. 

(c) Administered the written expression measure. 

(d) While the student was writing, scored and graphed the 
spelling and reading measures. 

(e) Scored/graphed the written expression measure at their 
convenience. 
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On days when four tasks were administered, the teachers: 

(a) Administered the reading passage measure, but did not 
score/graph at that time. 

(b) Administered the reading lists, but did not score/graph 
at 'that time. 

(c) Administered the spelling measure, but did not score/ 
graph at that time. 

/ 

(d) Administered the' written expression measure. 

(e) While the student was writing, scored/graphed the reading 
and spelling measures. 

(f) Scored/graphed the written expression measure at their 
convenience. 

Teacher Survey 1, indicating their satisfaction with the prescribed 
order, was completed by the teachers on the day following the end of the 
experimental phase. Teacher Survey 2, indicating the actual order they 
currently were employing, was completed two weeks later. 
Results — — 

For each subject, an AB time-series analysis was' applied to the 
average minutes per task that each teacher spent daily in preparing for 
and scoring and graphing the measures. Each teacher f s daily total time 
was divided by the number of tasks administered and this average time 
per task was graphed. The time spent in the administration of the 
measurement tasks was not included as this time was held constant 
regardless of the order of measurement. Three of the ten possible 
subjects were excluded from this analysis due to an insufficient number 
of data points (0 or 1) in Phase A (where measurement tasks were given 
in any order the teachers chose). Replications across teachers were 
assessed to determine the generality of the effect of the prescribed 
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order on teacher efficiency. In addition, teacher satisfaction was 

—assessed- via -the -survey^ 

The graphs displaying average time per task for Phases A and B for 
each teacher were inspected by four IRLD staff members experienced in 
time-series data analysis. Figure 1 is one of the seven graphs inspected. 
Each staff member independently judged the difference between Phase A 
and B on each graph as an increase, decrease, no 'change, or undetermined. 
Reliability for judgments on each graph ranged from .5 (1 graph) to .75* 
(5 graphs) to 1.0 (1 graph). Reliability wae high r/erall, with three 
out of four judges agreeing on all but one graph. For four of the seven 
graphs, minutes per task decreased; two were judged 'as no change, and 
judgments on the seventh graph were evenly split between decrease and 
undetermined. In addition, the mean net median change of minutes per 
task from Phase A (3.27) to Phase B (2.06) yielded a 37% decrease (a 
difference of 1.21 minutes). 



Insert Figure 1 about here 



On Teacher Survey 1, all teachers indicated satisfaction with the 

prescribed order. On a 1 to 4 scale (1 =* very dissatisfied, 4 = very * 

o 

satisfied), all teachers rated their level of satisfaction as "satisfied. 11 
On Teacher Survey 2, in reporting the order of administration actually' 
used, all teachers indicated use of the prescribed order. 
Discussion 

For four of the seven subjects, judges agreed that the prescribed 
order increased teacher efficiency. For two teachers, no change occurred 
and for one teacher the judges did not agree on the effect of the change. 
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The average difference between minutes per task in Phase A and Phase B 
was 1.21 minutes. Based on these results, it can be concluded that the 
prescribed order did increase teacher efficiency. Therefore, in order to 
increase efficiency, teachers can use the written expression student 
response time to score and graph other measures. 

Teachers also reported that they were satisfied with the prescribed 
order and did indeed choose to follow that order after the experiment 
was over. Given the increase in efficiency and the teacher reported 
preferred and actual practice, the prescribed order appears to be a 
satisfactory change in the measurement proce< re. Nevertheless, the 
procedures remained time consuming for teachers. 

V 

Study 3: Efficiency of When Measurement Occurs 

Proc edure 

v 

For one week (Phase A), given one student, teachers administered 
the measurement tasks at the middle or end of the instructional period, 
and completed the Observation Recording Sheet and the Student Transition 
Recording Sheet each day (see Appendix). For the following week (Phase 
B), given the same student, teachers administered the measurement tasks 
as soon as the student entered the room, and again completed the Observa- 
tion Recording Sheet and the Student Transition Recording Sheet each day. 

On the day following the last data collection time, teachers 
completed the teacher satisfaction survey (see Survey 2 in Appendix). 
Two weeks following that, teachers completed a survey indicating 
when, within the instructional period, they typically administered 
the measurement task (see Survey 3 in the Appendix). 
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Result s 

' o 

< 

For each subject, an AB time-series analysis was applied to three 
of the dependent measures: (a) daily total teacher time in preparing 
for, implementing, and scoring and graphing the measures; (b) estimated 
daily student transition time; (c) estimated number of daily teacher 
prompts to student while making the transition from measurement to 
normal routine. Two subjects were dropped due to insufficient data. ' 
Replications across teachers were assessed to determine the generality 
of how the factor, when the measurement occurred, affected teacher effi- 
ciency and student transition efficiency. Additionally, the two surveys 
were summarized across teachers to provide descriptive data on teacher 
satisfaction. 

For each teacher, efficiency data (total time in measurement) and 
student efficiency data (nu mber of minutes in transition plus number of 
verbal prompts) were displayed on a graph from which the subject name 
was removed. (See Figure 2 for an example of one of these graphs.) 
Then IRLD staff, experienced in time-series data analysis, independently 
inspected each graph, and judged by which of the following categories 
teacher total, time in Phase B was best described: increased, decreased, 
unchanged, or undetermined- Reliability across judges was high, with 



at least. three of the four IRLD staff concurring on all but two of the 
graphs. Total teacher time in Phase B (measurement at the beginning of 
the period) decreased for only one subject and increased for three of 
the subjects. There was no change for two cases; for the other two 
subjects, judgments were split between two categories. 



Insert Figure 2 about here 
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For each teacher the median estimated number of minutes the 
student spent transitioning to the next activity was calculated over 
Phase A and over Phase B; and for each teacher the net change in 
medians from Phase A to B was determined. Then, the net change was 
averaged across the teachers with a mean decrease of .31 estimated 
minutes. The number of verbal prompts from the teacher to the student 
in transition was summarized in an identical procedure, with a mean net 
median increase of .06 prompts. 

On Survey 2, 5 of 10 teachers indicated that they preferred to 
measure as soon as the student arrived. One teacher reported that she 
preferred to measure part way through the student's time in the resource 
room. Three teachers responded that they preferred measurement at the 
end of the student's instructional period. One teacher did not respond. 

On Survey 5, four teachers- reported that they-actually-measured at 
the beginning of the period; tv;o at the middle; none at the end. One 
teacher reported a variable schedule. 
Discussion 

For half the subjects, independent judges agreed that the variable 
"measuring at the beginning of the instructional period 11 had a controlling 
effect on teacher efficiency (operationalized as total time in measurement 
activities). Across tKose four subjects, however, the natme of that 
effect V as unclear, with a decrease for one teacher and increases for 
three of the teachers. Given these inconsistent results along with 
'the fact that there was either no change or the data were unclear for 
the other four subjects it appeared that, across teachers, measuring at 
the beginning of the period did not systematically control teacher 
efficiency. 

21 



17 

Small mean net median changes from Phase A to Phase B for (a) the 
, estimated nuriber of minutes the student spent transitioning to the next 
activity (-.31), and (b) the number of verbal prompts from the teacher 
to the student during transition (+.06) corroborated the fim'ing that 
measuring the student at the beginning of the period did not system- 
atically affect efficiency in measurement. 

9 

Teachers most frequently reported, however, chat they both preferred 
and actually implemented imeasureiuent at the beginning of the period. 
Therefore, teachers appeared more satisfied with measuring at the begin- 
ning of the instructional session. 

Study 4: Single Case Efficiency Studies 

Procedure 

In a workshop setting, teachers were presented with the results 
of Studies 2 and 3. Teachers and IRLD staff then discussed changes 
that could be made in the measurement procedures to increase efficiency. 
A list of possible changes was generated, including: 

(1) Use of tape recorders 

(2) Use of computers 

(3) Use of language master 

(4) Use of a peer to conduct the measurement 

(5) Ucc of a volunteer to conduct the measurement 

(6) Different material preparation and organization schemes 

(7) Using the student or a peer to score and graph data 

(8) Using peer administration for spelling measurement 

(9) Different procedures for selecting random samples as the 
stimuli for measurement 

(10) Using students to prepare and organize materials 
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(11) Using students to score the correct and incorrect letter 
sequences in spelling 

(12) Scoring a r .d graphing only correct responses rather than 
correct and incorrect responses 

After the list of changes was generated, teachers each chose a 
different factor to investigate. Eight single case studies were com- 
pleted, each using an ABA reversal design. Each "phase lasted approxi- 
mately two weeks during which time approximately six data points were 
collected. The implemented treatments are listed in Table 2. 



Insert Table 2 about here 

Data Analysis 

For each of the eight case studies, five categories of teacher 
eff *~iency were graphed: preparation time without student present; 
preparation time with student present; directions; scoring/graphing; 
and total time. For four of the cases (B, C, G, H), total time included 
an additional category of teacher efficiency, administration time. xQjis 
category was not included in other studies because administration time 
was held constant (one minute .for spelling, reading in context, and 
reading isolated woids) and therefore was not affected by the effi- 
ciency factors. However, in studies B, C, G, and H, the treatments 
eliminated teacher time spent in administration so the one minute ad- 
ministration time was included in the total time baseline phases. 

Each teacher's self-observed timings for each of the categories 
i 

* c . , 

were plotted for the three phasjes. Four staff members of the IRLD, 

experienced in time -series analysis, judged each treatment phase as 

effective, ineffective, or unclear on the five categories of teacher 
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efficiency for each of the eight subjects. Therefore, each rater made 
40 decisions. Interrater reliability was computed for the four judges 
by dividing the number of agreements by the number of possible agree- 
ments. Of the 40 cells, reliability was 100% for 28 cells, 75% for 
seven cells, and 50% for five cells. In other words, 70% of the judg- 
ments were identical; for all judgments, at least half of the raters 
agreed. ^ 
Results 

Efficiency. In case studies A, B, C, and D, the change in measure- 
ment procedures clearly reduced teacher time. (See Figures 3-6 for 
graphs of these studies.) 

Insert Figures 3-6 about here 



In the first modification of the measurement procedure (Study A), 
the teacher precounted the. number of words in the section of the book 
the student read for measurement purposes as opposed to counting the 
words read after the administration. This change in procedures increased 
the teacher's time spent in preparation without the student present from 
O^to 35 seconds but ( decreased time spent scoring/graphing from a median 

t 

of 50 seconds per task to 2 seconds per task. Total teacher time in 
measurement, excluding the one minute administration, was reduced from 
a median of 62.5 seconds per task to 44.5 seconds per task. 

In Study B^a^second modification in measurement procedures that 
reduced teacher time was investigated; specifically, the teacher taught 
the student to score and graph the Reading in Context and Reading in 
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Isolation measures. The time spent in instructing the student in 
these procedures was recorded under the scoring/graphing catego 
Initially, scoring/graphing time increased , dramatically , but then 
returned to a level as low or lower than the median time recorded 
before the change. Total time dropped from a median of 87 seconds 
per task to 47. Upon return to baseline (teacher scoring and graphing) 
median total time increased to 58- seconds per task. 

In the final two studies in which teachers clearly increased effi- 
ciency, both teachers used mechanical devices to administer the measure-^ 

ment tasks. In Study C, the- teacher used a language master to administer 

♦ *■ • 

the spelling measure individually to three children. During baseline 

phases the teacher had personally administered the spelling taeasure to 

each of the three students; each administration taKing one minu& per 

» 

student. During the treatmer<c phase, each child listened 'to the spelling 
words via the language master and therefore, the teacher saved the one 
minute administration time per student. The median total time for each 
phase was 130 seconds (teacher administration), 56 seconds (language 
master), and 85 seconds (teacher administration), respectively. 

The mechanical device used in Study D was a tape recorder. 
During the treatment phase, .instead of reading* a passage to the 
teacher, the student read for one minute into a tape recorder. With • 
this change in the measurement procedure, the teacher saved the one 
minute administration time. The median total time. per task for the 
three phases was 139.5 seconds (teacher listens), 88 seconds (tape 
recorder listens), and 73 seconds (teacher listens). The change in 
procedures cut the tot^ measurement time in half; when the teacher 




returned to baseline, there appeared to be some carryover effect. 

In study E, although* the results were less clear, it appeared 
that efficiency -was increased (see Figure 7). In this case, the change 
consisted of administering the spelling measure to two children at 

i 

once instead of individually. Median total time was reduced through 
each of the three phases from 188 seconds in Phase A (individual ad- 
ministration) to 137 seconds in Phase B (small group administration) , 
and then back to 59 seconds in Phase A (individual administration). The 
carryover of the treatment effect when returning to baseline made the " 
results difficult to interpret. 



, Insert Figure 7 about here 



In case studies F and G, the change in the measurement procedures 
involved the use of a peer tester or an aide to administer the measure- 
ment tasks. Neither study resulted in increased efficiency. In both 
cases, teachers spent an inordinate amount of time training the peer 

or the aide to conduct measurement. Preparation time with and without 
C 

student present and scoring/graphing time were increased during the 
treatment -phase. Therefore, even though the teachers saved themselves 
one minute of administration time, the increased time in the other 
categories negated any maximization of efficiency. For the study in- 
volving the peer tester, the median total times per task were 160, 
209, and 160 seconds foo: the three phases, indicating reduced effi- 
ciency. Median total time per task stayed fairly consistent across 
phases (110, 120, and* 118 seconds) in the study conducted with the 
aide as the measurement administrator. 
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In the final case study (H), the teacher switched from daily pre- 
paration of measurement materials during baseline to preparing the 
measurement materials for the entire week at one time. The only cate- 
gory of teacher time that could be affected directly by this change 
wa3 preparation without student present. However, the median times 
(40, 46, 45) reflected no difference in this category or in any other, 
including total time. This change in preparation strategy did not 
affect the teacher's efficiency. 

To summarize, in five of the eight case studies, teacher effici- 
ency increased. The changes in measurement procedures that increased 
efficiency included: (a) precounting oral reading passages; (b) using 
mechanical devices to administer the measurement tasks (language 
master for spelling and tape recorder for oral reading); (c) training 
the student to score and graph his/her own data; and (d) administering 
the measure to more than one student at a time instead of individually. 
• The three studies in which the changes failed to improve efficiency 
involved the use of v peer tester or aides in administering the measures 
and weekly, versus daily preparation of measurement materials. 

Teacher satisfaction . Teachers* were asked to respond to three 
questions about the changes made in measurement procedures: 

(1) How would you describe the effect of the change on your 
efficiency, your s'tudent(s), 6r any other factors? 

(2) What are the advantages of this change? 

(3) What are the disadvantages of this change? 

The teachers 1 responses are summarized in Table 2. Some of the ad- 
vantages mentioned by teachers included more time for instruction and 
increased motivation for the student. Disadvantages included increased 
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preparation time, student inconvenience, and less chance to conduct an 
error analysis of the student's performance when someone else adminis- 
tered the measurement task. 
Discussion 

The results of these case studies indicated that the teacher's time 
spent in measurement can be reduced. Procedures that seemed useful in 
increasing efficiency included precounting the words in oral reading 
passages, group administration, using mechanical devices to administer 
the measures, and teaching the student to score and graph measurement 
results. Using peer testers or aides to administer the measurement and 
preparing measurement material in bulk did not increase efficiency. 
However, if the treatment phases in these case studies had been expanded, 
increased efficiency may have been noted. In all three of these studies, 
the treatment phase consisted of only four or five data points, whereas 
the baseline phases ranged from 5 to 14 data points. In addition, dur- 
ing treatment a steep decreasing slope was noted in all cases. Had a 
longer treatment period been initiated, the desired effects might have 
been obtained. 

Study 5: A Foll ow Up of Teacher Efficiency 

Procedure 

For the remainder of the school year (February through May) , the 
10 teachers implemented whatever strategies they preferred in order to 
maximize their efficiency. In May, approximately nine months after 
initial training on the measurement procedure's, teachers measured 
their own behavior to determine how much time they typically spent in 
measurement activities. 
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Additionally, in January, a group of five suburban teachers was 
trained to prepare ; direct, administer, score, and graph the same 
measurement tasks by reading self-instructional manuals and by par- 
ticipating in several inservices., These teachers were observed twice 
in order to assess their efficiency. Unlike the procedure for the 
rural teachers, no training or systematic manipulations for improving 
efficiency were implemented. In May, approximately five months after 
they had been trained^in the measurement procedures, this contrast group 
of teachers monitored their own measurement activities to arrive at 
a time representative of their end-of-year efficiency. 
Results and Discussion 

The mean time per task for each teacher vas calculated. Then, 
for each group of teachers, these mean times were averaged, (In all 
cases, administration time was not included,) At the end of the year, 
the 10 rural teachers, who had been personally and intensely trained 
and systematically prompted in methods for improving efficiency, spent 
an average of 61 seconds in measurement for each task. In contrast, 
the group of teachers who had been trained via a manual and inservices, 
and instructed only verbally to improve their efficiency in whatever 
ways they cho^e, spent an average of 15 minutes per task in preparing, 
directing, scoring, and graphing. This year-end measurement time was 
much longer than the median measurement time obtained in January, which 
was two minutes and 21 seconds per task. Therefore, the unprompted 
group experienced a reduction in efficiency at the end of the year, 
performing 15 times less efficiently than the prompted group. 

It might be argued that the wide discrepancy between the times 
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of the two groups of teachers is a function of the difference in the 
number of months of practice rather than a function of the group 
different training histories. The prompted group spent an entire 
school year in measurement; the unprompted group spent only five months 
of the school year in measurement. However, comparing the two groups 
at the points where each had experienced a comparable four or five 
months in measurement, a large discrepancy remained. In January, the 
10 prompted teachers spent an average of 124 seconds per task in measure- 
ment activities; in May, the five unprompted teachers spent an average 
of 15 minutes per task. The prompted group was 600% more efficient 
than the unprompted group, even when the" number of months of practice 
time was held constant across the two groups. In fact, the unprompted 
group showed a decrease in efficiency time. 

Moreover, at the end of five months, the group trained by a manual 
and inservices was performing more than three times less efficiently 
(15 minutes) than the personally trained group performed initially 
(3 minutes, 26 seconds). This suggests that on-going, face-to-face 
training, along with specific instruction in methods for improving 
efficiency may be critical if teachers are to develop efficient measure- 
ment strategies.- , 

Obviously, there were other important differences between the 
groups of teachers that might account for the discrepancies. For ex- 
ample, the prompted group was rural while the unprompted group was 
suburban; the prompted group was required to participate while the 
unprompted group volunteered. Nevertheless, the prompted group's 
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performance throughout the year provides clear evidence that properly 
trained and prompted teachers can improve their efficiency dramatically. 
On the first trial the prompted group of teachers spent a median of 4 
minutes, 26 seconds in measurement per task; after two trials they 
spent 2 .minutes, 30 seconds, and by the end of the school year, that 
time had been reduced to 1 minute, 1 second* 

Conclusion 

In this research, teachers demonstrated that they reliably measure 
the time in which they are engaged in measurement activities. The 
analysis of their observations revealed that teachers initially required 
approximately 13 1/2 minutes to prepare, administer, and score and 
graph measurement tasks on four academic behaviors for one student. 
Given frequent measurement and the number of children typically on their 
caseloads, this time commitment becomes quite burdensome. As a result, 
an increase in the efficiency of the measurement procedures is critical 
if frequent measurement within the classroom is to be more feasible. 

Therefore, Studies 2 and 3 and a series of single case studies 
(Study 4) investigated the effects of various factors on the efficiency 
of measurement and teacher satisfaction. The first factor (Study 2) 
was a prescribed order consisting of the administration of the read- 
ing and spelling- tasks prior to the written expression task. This 
sequence allowed^the teachers to use the three minutes of student 
response time for written expression to score and graph the reading 
and spelling measures. Prior to the experimental phase, teachers had 
administered the tasks in whatever order they preferred and typically 
had been unoccupied while the student responded to the three minute 
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written expression measure. The prescribed order did increase slightly 
the efficiency of measurement and was satisfactory to the teachers. 

The second factor (Study 3), the scheduling of the measurement 
tasks, required the teachers to change from a phase in which they 
measured the students at the middle or end of the period to a phase 
in which they measured at the beginning of the period. Although no 
clear results were reported for efficiency, teachers v-re more satisfied 
with the latter schedule. 

The results of the single case studies (Study 4) suggested that 
other factors may also be effective in increasing teacher efficiency. 
Among the most effective factors were precounting the words in oral 
reading passages, group administration, using mechanical devices to 
administer the measures, and teaching the student to score and graph 
measurement results. Further investigations of these as well as other 
efficiency factors appear warranted. 

Study 5 contrasted these teachers' efficiency with the efficiency 
of a group of teachers who had been trained via a self-instructional 
manual and periodic inservices and who had been less thoroughly 
prompted to improve their efficiency. Results of Study 5 indicated 
that personal training along with systematic training in efficient 
procedures may be critical for developing efficient teacher measurement 
behavior. 
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Table 1 

Teacher Time in Administering SMT in Simulated and Natural Conditions 



Simulated 3 Natural b Simulated 0 Natural d 

(Week 1) (Week 2) (Week 3) (Week 4) 



Directions + Measurement Task 
+ Scoring/Graphing 

Reading Passages 

Range 



Preparation Time with 
Student Present 



Total Time vith 
Student Present 



Prep, »atlon Tiae without 
Student Present 



a N « 6 
b N - 10 



V25" - 2'45" 1'18" - 2' 16" 1»30" - V55.5" V29" - 2 , 01" 



Median 1'55" i'48 w 1'46.5" 1'47" 

Reading Lists 



Range l»20" - 3*56" 1'16 M - 2 , 47 n l^l.S" - l^.S" 

Median l'Al.S" 1'58.5" 1 , 44.3" 



1'20" - 2 , 02.5 M 
1'49.5 M 



Spelling 

n 

Range 2 , 55" - 6*55" 1'3" - 4 , 10" 2'23" - l'42" - 2'51" 

Median 2 , 50" 2'42" 3'06.5" 2 , 32" 
Written Expression 

Range 3'27" - 7*55" 3'23" - 7*5" V37.5" - 4 , 29.5" 3'39.5 - 4'55.5" 

Mcdian . , 4 , 37.5" 4'52.5" 3'58" 3'5l» 



0 , 0"-5 , 32" O'O" - 4 , 35 n O'O" - 0'60" 0'52" - 1'37.5" 

**dUn 2'9" 1/17" 0'24" 1'16" 



ll 9 3Z" - 2X , 52 n 7'24" - 17 , 25" 9 , 12" - 11'6" 8'54" 
Median XS'W 12 , 26.5" 



8*54" - 13'30" 



0 , 24" 



11* 18" 



***** O'O" - lO^" 2'X. 5" - 3*1. 4" 0'50" - 3'34" l'54» 



Median 4»15» 2*20. 4" 2 , 36" 

Total Time 



1'54" - 5' 02" 
2' 05" 



Range 14 , 32" - 22 , 12" 11*48. 8" - 16*48. 8" 9'54"- 12*20" n'19" - 16'20" 

Median 21 , 42" 13 , 47,5" 10 , 54" 13'36" 



c 

Entries are based on different subjects and different numbers of subjects (max N ■ 10). Subjects were 
assigned to prepare and administer only two measurement tasks. The times indicated are an average of the 
subject's and observer s times. 

"N.7 
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Table 2 

Summary of Single Case Studies 



Study 



Change 



Effect on 
Efficienoy 



Teacher's 
Response 



Advantage. 



s Disadvantages 



A Baseline - words counted during 
scoring/graphing 
Treatment - words precounted 
in reading selections 

B Baseline - teacher scores and 
graphs reading ceasures 
Treatment - student does own 
scoring and graphing for 
Reading 'in Context and 
Reading in Isolation 



Increased 



Increased 



Quicker to score 
and graph 



Too less time 



None 



Student was moti- 
vated 



Student liked to 
see data 
plotted on 
graph 



More cime 
corns uming 



Baseline - teacher individually 
administers spelling measures 

Treatment - language master 
used to administer spelling 
tasks 



Increased 



Increased effi- 
ciency 



Provided oppor- 
tunity for in- 
creased time in 
1 to 1 instruc- 
tion 



Required more 
preparation 



Baseline - student reads to 

tet.^er lor reading in context 

measurement 
Treatment - student reads into 

tape recorder for Reading 

in Qontext 



Increased 



Decreased effi- 
ciency , 



Allowed more class 
time with stu- 
dents 

Some motivating 
effect 



Time spent after 
school in 
scoring/graphing 
increased . 



Baseline - measure students 

individually 
Treatment - measure two 

student* at once in spelling 



Baseline - teacher conducts 

measurement 
Treatment - aide administers 

«««4ures 



Baseline - teacher conducts 

measurement 
Treatment - peer tester 

administers measures 



Mixed 



One student had 
to wait while 
other was still 
spelling 



Noue 



No change 



Not very efficient 



Teacher did not 
have to take 
time to adminis- 
ter measures 



Decreased 



More efficient 



Motivated 
student 



Student had to 
wait for the 
other to cstch 
up 



Required teacher 
cime to explain 
procedures to 
aide 



Less chance to 
see student er- 
rors 



Baseline - daily measurement 
preparation and organization 

Treatment - measurement mater- 
ials prepared once weekly 



No change 



No change 



None 



None 
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100- 



90 - 



80- 



70 



60 



50 - 



40 - 



30 - 



20 - 



10 - 



Task adminis- 
tration in 
middle of 
session 




LS7.VC 



(1)6 



« 8 


T 


■ 1 


10 


10 


10 


15 



Task adminis- 
tration at 
beginning of 
session 




number minutes 
for student to 
transition to 
next task 

number verbal 
prompts from 
teacher to 
student in 
transition 



10" 10 10 10 
20 21 22 23 



SCHOOL DAYS 



Figure 2. Time-Series Data for Subject 8, Study 2 



37 



• Preparation without 

student present 
K Preparation with 
stud mt-present 



• Directions 
X Scoring/graphing 




Figure 3. Time-Series Data for Single Case Study A 
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Figure 4. Time-Series Data for Single Case Study B 
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Figure 5. Time-Series Data for Single Case Study C 
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Student Transition Recording Sheet 
Measure the same student for these two weeks. 

Estimate how long it took the student to begin a new activity after 
you finished measuring him/her. Circle one. 



less than 1 min. 1 min. 2 min. 3 min* 4 min. 5 min. 6 min. 

7 rain. 8 min. 9 min. 10 min. more than 10 min. 



Estimate how many times you reminded the student to begin h? '*er new 
activity. 



(# of times) 



When within the instructional period did measurement occur? Circle one. 



beginning 
middle 
end 
other 



4'J 



Teacher Survey 1 



1. During this past week, while timing how long it took you to administer . 
the kit, please provide the following information: 

How many times did you administer each 
of these tasks? 



Student's name Spelling Reading Reading Written 

Passages Isolated Expression 
Words 



2. How easy is, it to time yourself while administering the kit? 
\ Very difficult Difficult Easy Very Easy 

3* In administering the kit last week, ratevybur level of satisfaction 

with the prescribed order. ■ : 1 -v^f 

Very dissatisfied Dissatisfied \E Satisfied Very satisfied 

1 2 3.4 

To what degree did each of the following factors contribute to 
your level of satisfaction? 

Low High 

12 3 4 How the prescribed order affected your speed in 
administering the tasks. 

12 3 4 How the prescribed order affected your accuracy in 
scoring the tasks ♦ 

12 3 4 How the prescribed order nffectcd the amount of 
time the student spent waiting. 
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Teacher Survey 2 



1. During this past week, while timing how long it took you to adrainist* 
the kit, please provide the following information: 

How many times did you administer each 
of these tasks? 

Student's name Spelling Reading Reading Written 

Passages Isolated Expression 
Words 



2. When you administer spelling and reading isolated words, what order 
of administration do you usually use? 



First 



Second 



3. When you administer spelling, reading isolated words, and written 
expression, what order of administration do you usually use? 



First 



Second^ 
Third* 



4. When you administer all four measures (spelling, reading isolated 
words, written expression, and reading passages), what order of 
administration do you usually use? 



First 



Second^ 
Third_ 
Fourth 
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Teacher Survey 3 



1/ Please provide the following information about how many tasks you 

administered from the kit to the research student in the past two weeks: 



How many times did you administer each 
of these tasks? 



Student's name 



Spelling Reading Reading 



Written 



Passages Isolated Expression 
Words* 



2. Please provide the following information about* when you administered 
the measurement tasks during the past two weeks ending 11/5 (either 
from the kit or your own materials) and the IEP. are^s in which "you 
are measuring and instructing: 

How many times did you adminis- 
ter a measurement task at each 
of these times? 



Student's name 



IEP areas of measurement 
and instruction 



Beginning 
r of the 
period 



Midddle End 
of the of the 
period * period 



3. Rate the usefulness of ttie measurement results that you have collected 
from the students. 



very 
useful 



moderately 
useful 



somewhat 
useful 



not at all 
useful 



A. How have you used the information that you collected 'through 
measurement? 



er|c 
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Teacher Survey 3 - cont. 



5. Assuming that your students had IEP goals in each of the measurement 

th r c e % S ; P i ea ;V ate y ° Ur 8atisfac «°» ^th the followinTfearures o? 
the Standard Measurement Kit. 

Ver y • Very 

Satisfied Satisfied Dissatisfied Dissatisfied 



The packaging and organization 1 

of the kit i 

The clarity of the directions 1 

The schedule of administration 1 

The spelling measurement format 1 

The written expression 

measurement format i 

The reading-in-con text 

measurement format i 

The reading isolated words 

measurement format l 



2 
2 
2 
2 
2 



3 
3 
3 
3 
3 



4 
4 
4 
4 
4 



4 
4 



f?r,t S ™n U ^ di 2 T U flnd UaVing P- re P ared measurement materials for the 
first month of data collection? 



Very 
Useful 



Moderately 
Useful 



Somewhat 
Useful 



Not at 
all Useful 



1 234 
7. Hov do you think the kit should be modified? 



8. With what Pspect(s) of the kit were you most pleased? 
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Teacher Survey 3 - cont. 



9. When do you prefer to administer the kit to the student? 



_As soon as the student arrives. 

/ 



Jart way through the student »s time in the resource room. 
_At the enu of the student's time in the resource room. : 



Teacher Observation Recording Form Name 



Date 



PREPARING 
WHILE STUDENT 
IS NOT PRESENT 



TIME: 



[#STUS: 



TIME: 



] [#STUS: 



TIME: 



[#STUS: 



TIME: 



[#STUS: 



TIME: 



] [#STUS 



TIf€: 



] [#STUS: 



PREPARING 
WHILE STUDENT 
IS PRESENT 



TIME: 



[#STUS: 



TIME: 



] I#STUS: 



TIME: 



] t#STUS: 



TIME: 



[#STUS: 



TIME: 



] [#STUS: 



TIME: 



] [#ST0S: 



GIVING DIRECTIONS 



READING IN READING IN 
CONTEXT CONTEXT 



SPELLING WRITTEN 
EXP RESSION 

TIME: TIME: 

I#STUS: ] [#STUS: ] [#STUS: ] [#STUS: ] 



TIME: 



TIME: 



SCORING & GRAPHING 



TIME: 



TIME: 



TIME: 



TIME: 
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